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ABSTRACT 

This  paper  deals  with  a new  quantitative,  vision-based  approach  to  road  following.  It  is 
based  on  the  theoretical  framework  of  the  recently  developed  optical  flow-based  visual 
field  theory.  By  building  on  this  theory,  we  suggest  that  motion  commands  can  be  gen- 
erated directly  from  a visual  feature,  or  cue,  consisting  of  the  projection  into  the  image 
of  the  tangent  point  to  the  edge  of  the  road,  along  with  the  optical  flow  of  this  point. 
Using  this  cue,  we  suggest  several  different  vision-based  control  approaches.  There  are 
several  advantages  to  using  this  visual  cue:  (1)  it  is  extracted  directly  from  the  image, 
i.e.,  there  is  no  need  to  reconstruct  the  scene,  (2)  for  many  road  following  situations 
this  is  the  only  necessary  visual  cue,  (3)  only  the  horizontal  component  of  the  optical 
flow  of  the  tangent  point  needs  to  be  extracted,  (4)  it  has  a scientific  basis,  i.e.,  the 
described  techniques  are  not  ad  hoc,  (5)  the  related  computations  are  relatively  simple 
and  thus  suitable  for  real-time  applications.  For  each  control  approach,  we  derive  the 
value  of  the  related  steering  commands. 


2 


- 3 - 


1.  INTRODUCTION 

Autonomous  visually-guided  road  following  by  a ground  vehicle  requires  two 
basic  steps.  The  first  step  is  to  extract  relevant  road  features  from  images  taken  by  on- 
board cameras.  This  step  involves  finding  road  regions,  road  edges  and  boundaries, 
center  lines,  highway  lane  lines,  etc.  in  complex  real-world  images.  The  second  step  is 
to  determine  how  to  steer  the  vehicle  (as  well  as  how  to  speed  up  or  slow  down)  once 
visual  road  information  has  been  extracted.  Algorithms  for  both  steps  have  recently 
been  explored  by  many  investigators  [2,  3,  4,  9,  10,  11].  Most  of  these  investigators, 
however,  have  approached  these  problems  using  ad  hoc  techniques.  That  is,  the  deter- 
mination of  which  road  features  to  extract,  as  well  as  how  to  use  them  to  make  steering 
decisions,  has  been  done  in  an  ad  hoc  manner. 

This  paper  approaches  the  road  following  problem  by  building  on  the  theoretical 
framework  of  the  recently  developed  visual  field  theory  [7,  8].  This  theory  provides 
quantitative  relationships  between  a stationary  3-D  environment  and  a moving  camera. 
The  theory  involves  pre-computing  the  expected  instantaneous  optical  flow  values  in 
the  camera  imagery  arising  from  every  point  in  3-D  space.  The  theory  provides  a 
theoretical  and  scientific  basis  for  optical  flow-based  road  following  algorithms. 

This  paper  is  concerned  with  an  analysis  of  the  road  following  problem.  Using 
the  visual  field  theory,  we  develop  geometric  and  motion-related  relationships  and  con- 
straints for  road  following.  These  then  suggest  partial  control  algorithms  as  well  as 
visual  cues  that  can  serve  as  input  signals  to  control  algorithms.  Rather  than  suggesting 
that  one  control  approach  is  superior  to  another,  the  paper  details  many  different  con- 
trol options.  Any  one  of  these,  or  some  combination,  can  be  used  in  the  development  of 
the  actual  control  algorithms.  The  control  schenxs  presented  are  partial  since  only  the 
kinematics  of  the  vehicle  and  the  camera  are  considered.  Also,  stability,  robustness  and 
sensitivity  issues  are  not  considered  in  this  paper.  Some  of  the  visual  cues  suggested  in 
this  paper  have  been  used  to  develop  road  following  control  algorithms  for  a real 
mobile  robot  [12].  Further  experiments  are  planned  in  the  future  to  validate  the  control 
schemes  presented  here. 

This  paper  has  suggestions  for  algorithms  for  both  steps  described  above.  For  the 
first  step,  involving  the  extraction  of  road  features,  this  paper  suggests  that  the  only 
road  feature  necessary  (for  curved,  convex  roads)  is  the  tangent  point  on  the  road  edge 
(i.e.,  the  point  on  the  road  edge  lying  on  an  imaginary  line  tangent  to  the  road  edge  and 
passing  through  the  camera)  and  its  optical  flow.  Therefore,  all  image  processing  effort 
should  be  directed  towards  reliably  finding  and  tracking  the  tangent  point  and  extracting 
its  optical  flow.  This  is  in  contrast  to  most  current  systems  which  attempt  to  find  as 
much  about  the  road  as  possible.  Existing  systems  often  ignore  the  tangent  point  when 
making  steering  decisions,  and  usually  are  not  concerned  with  the  optical  flow  values 
of  points  on  the  road. 

For  the  second  step  that  involves  determining  steering  commands  once  visual 
information  has  been  extracted,  this  paper  suggests  that  fast,  computationally  inexpen- 
sive, and  simple  control  algorithms  can  be  used.  Given  the  desired  distance  from  the 
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edge  of  the  road,  the  only  visual  information  these  algorithms  would  require  is  the 
location  of  the  tangent  point  (in  the  image)  and  its  optical  flow.  Other  inputs  to  these 
algorithms  would  be  the  current  steering  angle,  the  current  vehicle  speed,  etc.  The  out- 
put of  these  control  algorithms  is  the  change  in  steering  angle  for  the  next  instant  of 
time.  (In  this  paper,  we  do  not  consider  decisions  about  speed  and  acceleration  of  the 
vehicle.)  Most  existing  road  following  algorithms  convert  the  information  extracted 
from  images  into  a 3-D,  vehicle-centered  cartesian  coordinate  system  aligned  with  the 
ground  plane.  Steering  decisions  are  then  determined  in  this  coordinate  system.  A 3-D 
reconstruction  is  therefore  performed  before  steering  decisions  are  made.  We  suggest 
that  control  algorithms  can  be  developed  which  directly  use  image  information 
represented  in  the  2-D  image  coordinate  system  (e.g.,  image  position  and  image  flow). 
There  are  two  main  advantages  to  such  approaches.  The  first  is  that  since  3-D  recon- 
struction results  in  inferred  quantities  (i.e.,  recovering  the  third  dimension),  control  that 
is  based  on  this  method  is  not  as  robust  as  control  based  on  directly  observable  2-D 
image  quantities  [1].  Second,  control  based  on  2-D  image  quantities  is  simpler  and 
requires  much  less  computation;  it  is  therefore  much  faster. 

This  paper  begins  by  outlining  our  assumptions  and  defining  road  following,  the 
coordinate  system,  and  the  vehicle.  We  then  review  the  visual  field  theory  as  it  relates 
to  road  following.  Next,  we  provide  analyses  and  suggest  partial  control  schemes  for 
two  road  following  scenarios,  one  for  circular  roads  and  the  other  for  curved  roads. 
Finally,  we  discuss  directions  for  future  work. 

2.  DEFINITIONS  AND  ASSUMPTIONS 

2.1.  ROAD  FOLLOWING 

We  define  a road  as  any  continuous,  extended,  curvilinear  feature.  The  goal  of 
road  following  is  to  follow  along  this  feature  over  an  extended  period  of  time.  In  what 
we  normally  think  of  as  road  following,  a road  is  defined  either  by  its  boundaries  or  by 
an  extended  solid  or  dashed  white  line.  Here,  the  goal  is  not  only  to  follow  along  these 
features  but  also  to  stay  within  a constant  lateral  distance  from  these  features.  In  gen- 
eral, the  feature  to  be  followed  need  not  define  a real  road.  For  example,  it  could  be  a 
boundary  line  of  vegetation,  a stripe  painted  on  the  ground,  or  even  a wall.  For  a low- 
flying  air  vehicle,  the  feature  to  be  followed  could  be  a river. 

Vision-based  road  following  requires  the  ability  to  continuously  detect  and  track 
features  in  imagery  obtained  from  an  onboard  camera,  and  to  make  steering  decisions 
based  on  visual  properties  of  these  features. 

Figure  1 shows  a vehicle  on  a road  and  the  left-hand  side  road  edge.  The  unit  vec- 
tor h is  the  instantaneous  heading  of  the  vehicle,  0 is  the  instantaneous  center  of  curva- 
ture of  the  vehicle  path,  and  r is  the  instantaneous  radius  of  curvature  of  this  path.  We 
define  road  following  as  an  activity  that  involves  servoing  h such  that  it  follows  the 
road  edge.  It  is  desired  that  h be  servoed  such  that  the  vehicle  is  always  parallel  to  the 
tangent  to  the  local  curvature  of  the  road  edge  (Figure  lb),  and  such  that  the  distance  s 
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of  a point  on  the  vehicle  from  the  road  edge  is  maintained  at  a constant  value.  In  other 
words,  the  instantaneous  center  of  curvature  of  the  road  edge  and  the  instantaneous 
center  of  curvature  of  the  vehicle  path  should  coincide,  and  the  tangent  to  the  edge  of 
the  road  at  the  intersection  point  B should  be  parallel  to  h.  Normally,  these  constraints 
will  not  be  met  if  the  vehicle  is  attempting  to  avoid  an  obstacle  or  if  the  vehicle  is 
changing  lanes  on  a road.  If  a road  has  two  boundaries,  then  road  following  will  nor- 
mally involve  staying  within  these  boundary  edges.  In  this  paper,  we  assume  that  the 
road  is  always  curved  and  we  do  not  discuss  the  two-boundary  case. 

2.2.  COORDINATE  SYSTEM 

The  equations  in  this  paper  will  be  defined  in  a coordinate  system  which  is  fixed 
with  respect  to  the  camera  on  board  the  vehicle.  This  coordinate  system  is  shown  in 
Figure  2.  We  assume  that  the  camera  is  mounted  on  a vehicle  (later  we  explain  how) 
moving  in  a stationary  environment.  Assume  a pinhole  camera  model  and  that  the 
pinhole  point  of  the  camera  is  at  the  origin  of  the  coordinate  system.  This  coordinate 
system  is  used  to  measure  angles  to  points  in  space  and  to  measure  optical  flow  at 
these  points.  We  use  spherical  coordinates  iR-Q-<^)  for  this  purpose.  In  this  system, 
angular  velocities  (9  and  (j>)  of  any  point  in  space,  say  P,  are  identical  to  the  optical 
flow  values  at  P'  in  the  image  domain.  Figure  3 illustrates  this  concept:  0 and  4)  of  a 
point  in  space  are  the  same  as  0 and  4)  of  the  projected  point  P'  in  the  image  domain, 
and  therefore  there  is  no  need  to  convert  angular  velocities  of  points  in  3D  space  to 
optical  flow.  In  Figure  3 the  image  domain  is  a sphere.  However,  for  practical  pur- 
poses the  surface  of  the  image  sphere  can  be  mapped  onto  an  image  plane  (or  other 
surface). 

2.3.  TWO-WHEELED  VEHICLE 

For  our  analysis,  we  use  a theoretical  two-wheeled  vehicle  as  illustrated  in  Figure 
4.  A rigid  frame  of  length  2m  holds  both  wheels.  A steering  wheel  angle  is  applied  to 
both  wheels  simultaneously,  i.e.,  if  one  wheel  is  rotated  by  an  angle  p relative  to  the 
frame,  the  other  wheel  will  rotate  by  the  same  angle.  This  apparatus  assures  that  both 
wheels  will  always  stay  at  the  same  distance  from  the  instantaneous  center  of  curvature 
of  the  vehicle’s  path.  As  shown  in  Figure  4,  the  camera  is  mounted  such  that  its 
pinhole  point  is  located  above  the  front  wheel  center,  and  it  rotates  with  the  front 
wheel.  The  optical  axis  of  the  camera  coincides  with  the  instantaneous  translation  vec- 
tor (heading)  of  the  front  wheel.  Note  that  the  heading  vector  of  other  points  on  the 
vehicle  may  not  be  the  same  as  the  chosen  one. 

The  following  geometrical  relationship  holds  for  the  vehicle  in  Figure  4: 


The  frame  length  m is  usually  known.  Thus  the  instantaneous  radius  of  curvature  r of 
the  vehicle  path  can  be  determined  by  measuring  the  steering  angle  p. 
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Figure  5 is  an  overall  description  of  the  system  including  the  spherical  coordinate 
system,  and  Figure  6 is  the  related  top  view.  For  convenience  we  chose  to  have  the  Z 
axis  pointing  down.  However  the  same  coordinate  system  as  described  in  Figure  2 is 
used  here.  The  camera  is  mounted  at  some  height  above  the  ground  and  rotates  with 
the  front  wheel.  The  position  of  any  point  on  the  road  can  be  expressed  with  the  coor- 
dinates ^,0  and  ()),  as  shown  in  Figure  5. 

In  the  following  analysis,  we  assume  a moving  vehicle  in  a stationary  environ- 
ment. The  road  is  assumed  to  be  planar,  and  road  edges  are  assumed  to  be  extractable. 
Figure  7 is  an  example  of  a road  image  obtained  from  a camera  mounted  on  a vehicle. 

3.  VISUAL  FIELD  THEORY 

We  have  recently  developed  a new  visual  field  theory  that  relates  six-degree-of- 
freedom  camera  motion  to  optical  flow  for  a stationary  environment  [7,  8].  The  theory 
describes  the  structure  of  a field  in  3-D  space  consisting  of  contours  and  surfaces  sur- 
rounding the  moving  camera.  If  static  objects  are  placed  anywhere  in  the  surrounding 
space,  the  optical  flow  produced  by  these  objects  in  the  camera  is  predicted  by  the  field 
theory.  The  field  is  always  centered  at  the  camera  pinhole  point  and  moves  with  the 
camera.  The  structure  of  the  field  changes  as  a function  of  the  instantaneous  camera 
motion. 

This  theory  provides  us  with  a theoretical  and  scientific  basis  for  developing  con- 
straints, control  schemes,  and  optical  flow-based  visual  cues  for  road  following.  This 
section  reviews  this  theory  as  it  relates  to  the  road  following  problem. 

3.1.  EQUATIONS  OF  MOTION  AND  OPTICAL  FLOW 

First  we  describe  the  equations  that  relate  a point  in  3-D  space  to  the  projection  of 
that  point  in  the  image  for  general  six-degree-of-freedom  motion  of  the  camera.  Some 
of  the  equations  can  be  found  in  many  books,  e.g.,  see  [5]. 

We  start  with  the  derivation  of  the  velocity  of  a 3-D  point  in  the  XTL  coordinates 
(Figure  2).  Let  the  instantaneous  coordinates  of  the  point  P be  R = {X,Y jzf  (where  the 
superscript  T denotes  transpose).  If  the  instantaneous  translational  velocity  of  the  cam- 
era is  t = {Uy,wf  and  the  instantaneous  angular  velocity  is  (o  = (A  .5 ,Cf  then  the  velo- 
city vector  V of  the  point  P with  respect  to  the  xyz  coordinate  system  is: 


V = - t - © X R 

(2) 

or; 

tAf  = -U-BZ+CY 

(3) 

Vy  = -V-CX+AZ 

(4) 

^2  = -W-AY+BX 

(5) 

where  Vx,  and  Tz  are 

directions  respectively. 

the  components  of  the  velocity  vector  V along  the  X,Y,  and  Z 
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To  convert  from  /?0<j)  to  XYZ  coordinates  we  use  the  relations: 


X = R cos4)  COS0  (6) 

Y = R cos<l)  sin9  (7) 

Z = R sin()).  (8) 

Similarily,  to  convert  from  XYZ  to  /?04)  coordinates  we  use: 

R = <X'^+Y^+Z^  (9) 

8 = tan-‘|-  (10) 


In  order  to  find  the  optical  flow  of  a 3-D  point  in  R 04)  coordinates,  we  use  the  follow- 
ing relations  and  transformations  (see  [6]  and  Figure  2): 

Let  V/?,  Vq,  and  be  the  components  of  the  vector  V in  spherical  coordinates,  and 


- 


(12) 


^XYZ  - 


Vx 

Vy 

Vz 


Then: 


Vrb^  = [T,][T^]Vxyz 


where 


[lel  = 


COS0 

-sin0 

0 


sin0  0 
COS0  0 
0 1 


and 


[T^] 


cos4>  0 sin4> 
0 1 0 
_-sin4>  0 cos4>. 


Also  (see  [6]): 


0 

0 

1. 

VPTF 

0 

z 

<X^+Y^+Z^ 

0 

1 

0 

-z 

0 

iX^+Y^+Z^ 

Vx^+y^+z^ 

(13) 


(14) 


(15) 


(16) 
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Vr=R 


Vq  = RQ  cos4) 
V,  = R^ 


where  dot  denotes  first  derivative  with  respect  to  time. 

Using  equations  (3)-(19)  yields  the  following  expressions: 


^9  COS0 
^0 

- sin0  cos9 

L-  sin0  COS0  - sin0  sin9 

or 

-y 

X 

0 

x2+y2 

x^+y^ 

A 

-xz 

-yz 

1 

-U-BZ+CY 

-V-CX+AZ 

-W-AY+BX 

0 

X^+Y^+Z^ 


-U-BZ+CY 

-V-CX+AZ 

rW-AY+BX_ 


(17) 

(18) 
(19) 


(20) 


(21) 


As  mentioned  earlier,  0 and  4)  of  a point  in  space  (i.e.,  the  angular  velocities  in  the 
camera  coordinate  system)  are  the  same  as  the  optical  flow  components  9 and  0 (Figure 
3). 


Suppose  that  we  want  to  determine  the  locus  of  points  in  3D  space  that  produce 
constant  optical  flow  values  9 and  constant  optical  flow  values  0 in  the  image  for  a 
given  arbitrary  six-degree-of-freedom  camera  motion.  To  do  so  we  simply  set  9 and  0 
in  equation  set  (21)  to  the  desired  constants  and  solve  for  X,  Y,  and  Z.  All  points  in 
3-D  space  that  satisfy  this  solution  are  called  equal  flow  points.  However,  the  solution 
to  these  two  equations  is  not  unique  since  there  are  three  unknowns  and  two  equations. 
In  general,  there  is  an  infinite  number  of  solutions. 


3.2.  A SPECIAL  CASE 


In  this  section  we  analyze  a specific  motion  in  the  instantaneous  XY  (0  = 0)  plane 
of  the  camera  coordinate  system. 

Let  the  camera  motion  vectors  t and  to  be  given  as  follows: 

t = (t/,F,0)^  (22) 

Q)  = (0,0,0^.  (23) 


This  means  that  the  translation  vector  may  lie  anywhere  in  the  instantanous  XY  plane 
while  the  rotation  is  about  the  Z-axis.  Substituting  these  motion  vectors  into  equation 
set  (21)  yields: 


0 


-Y 

X^+Y^ 

-xz 

^x^+y\x^+y\z^) 


X 

X^+Y^ 

-YZ 

^/P+P(x2+y^+z^) 


0 

x^+y^+z^ 


-U+CY 

-v-cx 

0 


(24) 
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Setting  0 and  4>  in  equation  set  (24)  to  constants  will  result  in  a set  of  equal  flow  points 
for  this  specific  motion. 

Consider  the  case  where  the  optical  flow  value  of  0 is  constant.  From  equation  set 
(24),  the  points  in  space  that  result  from  constant  0 (regardless  of  the  value  of  0)  form 
a cylinder  of  infinite  height  whose  equation  is 


X+ 


V 


'I  2 


2(C+0) 


Y- 


U 


2(C+0) 


r 

V 

2 

u 

2(C+0) 

2(C+0) 

(25) 


as  displayed  in  Figure  8. 

Figure  9 shows  a horizontal  section  of  the  cylinder  of  Equation  (25).  The  section 
is  a circle  that  lies  in  the  XY  plane.  This  plane  is  perpendicular  to  the  axis  of  symmetry 


of  the  cylinder.  The  radius  of  the  circle  is 
V u 


F 

2 

r 

u 

2(C4-0) 

J 

2(C+0) 

^ J • 

and  its  center  is  at 


2(C+0)  2(C+0) 


. The  circle  is  tangent  to  the  camera  translation  vector  at  the  ori- 


gin. 


The  meaning  of  equation  (25)  is  the  following:  all  points  in  3-D  space  that  lie  on 
the  cylinder  described  by  Equation  (25)  and  which  are  visible  (i.e.,  unoccluded  and  in 
the  field  of  view  of  the  camera)  produce  the  same  instantaneous  horizontal  optical  flow 
0.  We  call  the  cylinder  on  which  equal  flow  points  lie  the  equal  flow  cylinder.  A sec- 
tion of  a set  of  equal  flow  cylinders  is  iUustrated  in  Figure  10  for  the  case  where  the 
camera  undergoes  instantaneous  translation  and  rotation.  The  label  of  each  circle 
represents  the  horizontal  optical  flow  0 in  the  image  that  corresponds  to  points  on  this 
circle.  Here,  there  is  a circle  with  finite  radius  that  produces  zero  horizontal  flow  (0  = 0 
in  the  image  domain). 


3.3.  ZERO  FLOW  CYLINDERS 


One  of  the  equal  flow  cylinders  corresponds  to  points  in  3-D  space  that  produce 
zero  horizontal  flow.  We  call  this  cylinder  a zero  flow  cylinder.  The  equation  that 
describes  the  zero  flow  cylinder  can  be  obtained  by  setting  0 = 0 in  Equation  (25),  i.e.. 
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We  have  shown  [7]  that  if  the  Z component  of  the  camera  rotation  vector  co  is  positive 
(i.e.,  C > 0),  then  visible  points  in  the  XY  plane  that  are  inside  the  zero  flow  cylinder 
produce  positive  horizontal  optical  flow  (0  > 0),  while  visible  points  outside  the  zero 
flow  cylinder  produce  negative  horizontal  optical  flow  (0  < 0)  in  the  image  (see  Figure 
11).  If  (D  is  negative  (i.e.,  C > 0)  then  the  opposite  is  true. 
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3.4.  EQUAL  FLOW  CYLINDERS  AS  A FUNCTION  OF  TIME 

As  the  camera  moves  through  3-D  space,  the  equal  flow  cylinders  move  with  it. 
Figure  12  shows  sections  of  equal  flow  cylinders  as  a function  of  time.  At  each  instant 
of  time,  the  radii  of  the  equal  flow  cylinders  are  a function  of  the  instantaneous  motion 
parameters  t and  co.  The  locations  of  the  equal  flow  cylinders  are  such  that  they 
always  contain  the  origin  of  the  camera  coordinate  system  (the  same  as  the  camera 
pinhole  point),  are  tangent  to  the  instantaneous  translation  vector  t,  and  their  symmetry 
axes  are  parallel  to  the  instantaneous  rotation  vector  co.  (In  Figure  12,  the  direction  of  o) 
varies  over  time.)  Each  zero  flow  cylinder  lies  to  the  left  or  right  of  the  translation  vec- 
tor depending  on  whether  the  instantanous  rotation  is  positive  or  negative,  respectively. 

4.  ANALYSIS  OF  ROAD  FOLLOWING 

We  describe  two  road  following  scenarios.  The  first  one  is  for  a circular  road, 
where  we  outline  basic  geometric  and  motion-related  relationships.  Using  this  rela- 
tively simple  case,  we  explain  the  problem  of  following  a road  using  a vision  sensor, 
problems  associated  with  it,  and  relate  it  to  the  visual  field  theory  described  above.  We 
also  suggest  several  road  following  control  approaches.  The  second  road  following 
scenario  is  for  an  arbitrary  convex  curved  road,  where  we  also  suggest  some  control 
approaches. 

4.1  CIRCULAR  ROAD 

In  this  section,  we  consider  following  along  a circular  road.  Given  visual  cues,  a 
goal  of  a control  system  is  to  find  the  steering  angle.  If  the  vehicle  is  already  on  a 
path  that  follows  the  road,  then  only  changes  in  steering  angle  are  necessary.  Figure 
13  shows  a vehicle  moving  around  a circular  road  of  radius  /.  The  path  traversed  by 
the  vehicle  is  a circle  of  radius  r.  Let  the  unit  vector  t indicate  the  direction  of  the 
tangent  line,  a line  that  contains  the  camera  pinhole  point  and  is  tangent  to  the  road 
edge. 

We  will  show  next  that  the  tangent  point  T lies  on  the  instantaneous  zero  flow 
cylinder  if  the  camera  orientation  is  fixed  relative  to  the  vehicle.  The  proof  for  this  is 
as  follows.  Let  us  consider  the  planar  case  first,  as  shown  in  Figure  13.  Given  that  the 
line  AT  is  tangent  to  the  road  edge,  then  AT  is  perpendicular  to  OT.  We  will  now  show 
that  OA  is  the  diameter  of  the  section  of  the  zero  flow  cylinder  displayed  in  Figure  13. 
From  Equation  25,  we  can  see  that  the  center  of  the  zero  flow  cylinder  section  is  at 

(Figure  9).  Further,  the  location  of  the  camera  is  at  (0,0).  Now  the 

center  of  rotation  of  the  camera’s  circular  path  must  lie  on  the  zero  flow  cylinder  sec- 
tion. To  see  why,  we  first  note  that  the  vector  from  the  camera  to  the  center  of  rotation 
0 is  always  perpendicular  to  the  camera  heading  vector.  (Remember  that  the  heading 
vector  and  camera  optical  axis  coincide.)  Therefore  the  center  0 can  be  considered  as  a 
fixation  point  for  the  camera  during  its  motion,  i.e.,  the  position  of  point  0 does  not 
change  relative  to  the  camera’s  coordinate  system  [7,  8].  This  means  that  this  point 
must  theoretically  produce  zero  horizontal  optical  flow  in  the  camera  (assuming  the 
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camera  has  a wrap-around  lens).  Therefore,  point  O must  lie  on  the  zero  flow  cylinder 
section.  Then  the  center  of  rotation  of  the  camera’s  circular  path  must  be  at 


V 


u 


(Figure  9).  Therefore  line  OA  is  the  diameter  of  the  zero  flow 

(C+e)  (c+G)^ 

cylinder.  Since  the  angle  OTA  is  a right  angle,  point  T must  lie  on  the  circle  whose 
diameter  is  OA.  Since  this  circle  is  a section  of  the  zero  flow  cylinder,  point  T must  lie 
on  this  cylinder.  For  the  case  where  the  road  edge  does  not  lie  in  the  X-Y  plane  of  the 
camera,  all  points  above  or  below  the  point  T (including  the  one  on  the  edge  of  the 
road)  lie  on  the  zero  flow  cylinder. 


Notice  that  this  proof  holds  no  matter  what  the  diameter  of  the  circular  road  edge. 
This  means  that  no  matter  how  far  the  vehicle  is  from  the  road  edge  (Figure  14),  the 
tangent  point  lies  on  the  zero  flow  cylinder.  Thus  the  horizontal  component  of  optical 
flow  of  the  tangent  point  is  always  zero. 

In  Figure  13,  therefore,  the  optical  flow  0 due  to  point  T is  zero.  Let  the  distance 
from  the  vehicle  to  the  road  edge  be  j,  and  let  9 be  the  positive  angle  to  t measured 
from  the  X-axis.  From  Figure  13,  the  following  relationships  hold; 


/ = r sinG 

s = r-l  = r(l-sin0) 

Differentiating  Equation  (27)  with  respect  to  time: 

/ = rsinG  + r GcosG 


(27) 

(28) 

(29) 


where  dot  denotes  derivative  with  respect  to  time.  For  a circular  road,  I is  constant, 
and  thus  / can  be  set  to  zero  in  Equation  (29): 

0 = rsinG  + rG  cosG 


r = -r  Q cot  9 (30) 

When  the  vehicle  is  moving  on  a perfect  circular  path  both  r and  9 are  equal  to  zero. 
However,  suppose  the  vehicle’s  path  is  not  a perfect  circle.  Since  r is  the  instantaneous 
radius  of  curvature  of  the  vehicle  motion,  r is  the  rate  at  which  the  curvature  changes. 
Equation  (30)  suggests  a way  of  controlling  the  vehicle  motion  so  as  to  achieve  a con- 
stant circular  motion.  Consider  the  two-wheeled  vehicle  described  in  Section  2.3.  From 
Equation  (1),  we  can  derive  the  following: 


[3  = sin  '(— ).  (31) 

r 

Equation  (31)  gives  a value  of  the  steering  angle  3 as  a function  of  the  instantaneous 
radius  of  curvature  r and  the  distance  2m  between  the  two  wheels.  Normally  the  value 
m is  known.  For  a more  realistic  vehicle  (e.g..  Figure  15),  some  other  relationship  may 
hold. 

In  Equation  (30),  r is  the  rate  at  which  the  radius  of  curvature  of  the  vehicle 
motion  is  changing.  We  can  express  r as  a function  of  the  steering  angle  p by 
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differentiating  Equation  (1)  with  respect  to  time: 

-m  cosB  A 

P 

sin^p 

Substituting  Equations  (32)  and  (1)  into  (30)  and  solving  for  p : 

P = 0 tanP  cotQ 


(32) 

(33) 


Equation  (33)  suggests  a partial  control  scheme  whose  inputs  are  the  current  steering 
angle  p,  the  current  angle  9 of  the  tangent  line  relative  to  the  X-axis,  and  the  optical 
flow  0 of  the  tangent  point.  All  of  these  inputs  can  be  measured.  The  variable  being 
computed  is  the  rate  of  change  of  the  steering  angle,  p.  Equation  (33)  provides  the 
gain  tanPcot9  by  which  0 should  be  multiplied  in  order  to  get  the  correct  change  in 
steering  wheel  angle.  This  gain  depends  on  the  current  steering  wheel  angle  p and  the 
angular  location  0 of  the  tangent  point  in  the  image.  Note  that  Equations  (1)  and  (31) 
hold  only  for  certain  types  of  vehicles.  Vehicles  with  other  wheel  and  steering 
configurations  (e.g..  Figure  15)  will  result  in  different  expressions  relating  steering 
angle  to  the  radius  of  curvature  of  motion.  In  all  such  expressions,  however,  there 
should  be  a one-to-one  relationship  between  p and  r.  These  expressions  can  then  be 
substituted  into  Equation  (30)  to  derive  the  relevant  control  signals.  It  is  important  to 
emphasize  that  the  derivation  of  p takes  into  account  the  kinematics  of  the  system  but 
not  the  dynamics.  This  is  also  the  reason  why  we  emphasize  that  the  control  scheme 
is  not  Complete. 

If  the  rate  of  change  of  the  steering  angle,  p,  is  the  only  variable  being  controlled 
(as  indicated  in  Equation  (33)),  then  in  practice  the  vehicle  may  not  maintain  a constant 
distance  from  the  edge  of  the  road.  Therefore,  in  addition  to  Equation  (33),  Equation 
(28)  can  also  be  used  to  control  the  vehicle  to  achieve  a constant  circular  motion.  Sub- 
stituting Equation  (1)  into  (28): 


or 


5 


p = sin“'[— (l-sin0)]  (34) 

s 

Equation  (34)  suggests  a partial  control  scheme  whose  inputs  are  the  measured  angle  0 
of  the  tangent  line  relative  to  the  X-axis,  the  desired  distance  5 of  the  vehicle  from  the 
road  edge,  and  the  distance  2 m between  the  front  and  rear  wheels.  The  variable  being 
computed  is  the  steering  angle  p. 

The  control  signals  (P  and  P)  and  partial  control  schemes  suggested  above  assume 
that  the  road  is  circular,  that  the  center  of  curvature  of  the  vehicle  path  coincides  with 
the  center  of  curvature  of  the  circular  road,  and  that  the  road  is  planar.  It  is  also 
assumed  that  the  tangent  point  (in  the  image)  is  traceable,  and  that  the  vehicle  heading 
coincides  with  the  camera  optical  axis.  There  are  several  advantages  to  this  approach: 
(1)  it  is  simple  and  therefore  computationally  inexpensive,  (2)  it  is  independent  of  the 
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speed  of  the  vehicle,  (3)  it  is  independent  of  the  camera  height  above  the  road,  (4)  only 
a few  measurements  are  necessary  to  control  the  vehicle,  and  (5)  only  a very  small  por- 
tion of  the  image  - the  portion  around  the  tangent  point  - needs  to  be  analyzed,  in 
principle.  (Of  course,  item  (5)  may  not  be  true  in  practice  since  larger  portions  of  the 
road  may  have  to  be  extracted  in  order  to  reliably  find  the  tangent  point.) 

A different  approach  for  circular  road  following  is  based  on  the  simple  fact  that 
the  height  of  the  camera  above  the  road  is  constant  during  driving.  Refer  to  Figures  5 
and  13.  Let  he  be  the  height  of  the  camera  above  the  ground.  For  the  tangent  point  T 
we  can  write: 


he  = r COS0  tan(J) 

or 

he 

r = 

COS0  tan<t) 

The  value  he  is  usually  a known  constant,  while  0 and  0 for  the  tangent  point  can  be 
measured  in  the  image.  Thus  the  distance  r of  the  camera  from  the  center  of  curvature 
of  the  circular  road  can  be  determined.  Equations  (36)  and  (31)  allow  computing  the 
steering  wheel  angle  p using  visual  information  without  measuring  it  directly. 

Taking  the  derivative  of  equation  (35)  with  respect  to  time  yields; 

he  = r COS0  tan0  + r (sinQ)  tan0  0 + 7-  cos0  — <}>  (37) 

cos  0 

Since  he  is  constant,  then  = 0 in  Equation  (37),  and  solving  for  r: 

f = (0  tan0 ^-i-)  r (38) 

sin20 

The  corresponding  change  in  steering  angle  is  (using  equations  (32)  and  (38)): 

P = — - sinP  tanp  (0  tan0 ^-^)  r (39) 

m sin20 

tn  ^ 

Substituting  p = sin"^-^)  (Equation  (31))  and  then  r = (Equation  (36))  in  Equa- 

tion (39)  results  in  an  expression  for  p as  a function  of  the  visually  measured  parame- 
ters 0,0,0,0  and  the  known  constants  m and  he. 

Yet  another  approach  for  circular  road  following  is  depicted  in  Figure  16.  A circu- 
lar path  will  be  maintained  by  the  vehicle  if  the  heading  vector  h^  at  point  A is 
smoothly  servoed  so  as  to  result  in  a heading  vector  fig  at  point  B . It  is  desired  that  the 
heading  vector  always  be  tangent  to  the  circular  path  concentric  with  the  circular  road 
and  at  a radial  distance  5 from  the  road  edge.  Let  0 be  the  angle  between  the  camera 

X -axis  and  the  tangent  line,  and  let  a = y - 0.  Notice  that  since  it  is  desired  that  hg  be 

parallel  to  the  tangent  line  from  point  A , the  change  in  direction  between  h^  and  fig  is 
a. 


(35) 

(36) 
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We  define  the  turning  rate  as  the  amount  the  heading  vector  direction  changes  (or 
turns)  per  unit  time. 

Since  the  circular  arc  length  between  points  A and  B is  (5  + / ) a,  it  takes  the  vehi- 
cle “ amount  of  time  to  travel  the  length  of  the  arc,  where  v is  the  speed  of  the 

vehicle  (assuming  constant  speed).  Therefore 

Turning  Rate  = 


a 


{s+l)a 


V 

s+l 


(40) 


For  the  two-wheeled  vehicle  in  Figure  4,  the  turning  rate  is  proportional  to  the  rate  of 
change  of  the  steering  angle,  i.e.,  p. 

4.2  CURVED  ROAD  FOLLOWING 

In  this  section,  we  consider  road  following  for  the  case  where  the  curvature  of  a 
convex  road  is  not  constant.  Figure  17  shows  two  cases.  In  Figure  17a  the  radius  of 
curvature  increases  as  the  vehicle  moves.  In  Figure  17b,  the  radius  of  curvature 
decreases.  Figure  18  shows  a detailed  version  of  Figure  17a.  Let  the  current  instan- 
taneous center  of  curvature  of  the  vehicle  path  be  at  0 . If  the  road  curvature  were  con- 
stant, then  the  point  of  tangency  of  the  vector  1 would  be  at  r and  this  point  would  lie 
on  the  zero  flow  cylinder.  However,  because  the  road’s  curvature  is  changing,  the 
point  of  tangency  is  at  T'.  The  center  of  curvature  of  the  curve  at  T'  is  at  O'.  Notice 
that  the  point  T'  lies  on  some  equal  flow  cylinder  whose  9 optical  flow  is  negative  (T 
lies  outside  the  zero  flow  cylinder).  If  the  radius  of  curvature  were  decreasing  (Figure 
17b),  the  tangent  point  would  lie  inside  the  zero  flow  cylinder,  and  its  0 optical  flow 
would  be  positive.  Therefore,  intuitively,  if  the  horizontal  component  of  the  optical 
flow,  0,  at  the  tangent  point  is  measured,  then  its  value  can  be  used  as  a control  signal 
for  steering  the  vehicle.  If  0 is  negative  (Figure  17a)  then  the  steering  command  is  to 
increase  the  radius  of  curvature  of  the  vehicle’s  current  motion.  If  0 is  positive  (Figure 
17b),  then  the  steering  command  is  to  decrease  the  radius  of  curvature  of  the  vehicle’s 
current  motion  by  sharpening  the  turn. 

Another  approach  to  curved  road  following  is  to  extend  the  technique  shown  in 
Figure  16.  Consider  Figure  19,  which  shows  the  case  of  a road  whose  radius  of  curva- 
ture is  increasing.  As  before,  the  goal  is  to  smoothly  servo  the  heading  vector  at 
point  A so  as  to  result  in  a heading  vector  ha  at  point  B . The  vector  hg  is  parallel  to  the 

tangent  line  from  point  A and  is  at  a distance  ^ from  the  line.  If  a = y - 9 is  the  angle 

between  and  the  tangent  line,  then  the  change  in  direction  between  and  hg  is  a. 
Let  the  distance  between  the  vehicle  and  the  tangent  point  r be  a.  Later  we  will  dis- 
cuss how  a might  be  computed.  The  arc  between  points  A and  B can  be  very  closely 
approximated  by  an  arc  of  a circle.  The  straight  line  distance  between  A and  B is 


Va^+P.  The  ratio  between  the  arc  AB  and  the  line  AS  is 


circle).  Then 


a 


2sin4 

2 


(assuming  arc  AS  is  a 
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Turning  Rate  = — ■ 


2 V sin( ) 

4 2^ 


(41) 


A way  to  compute  a in  Equation  (41)  is  as  follows.  Suppose  the  camera  is  located  at  a 
height  above  the  ground  (Figure  20).  The  angle  between  the  horizontal  and  the 

tangent  line  is  ()).  Then,  from  Figure  20, 


Since  angle  $ can  be  measured  in  the  image  and  h^.  is  a known  constant,  it  is  easy  to 
compute  a from  Equation  (42). 

Equation  (41)  gives  the  turning  rate  to  achieve  road  following.  The  quantities  that 
need  to  be  measured  to  achieve  this  are  the  angle  e between  the  X-axis  and  tangent 
vector,  the  value  a,  and  the  velocity  v of  the  vehicle.  The  value  5 is  assumed  to  be 
given. 

A somewhat  different  method  of  controlling  the  vehicle  can  also  be  derived  from 
Figure  19.  This  method  assumes  that  the  control  scheme  is  to  continuously  compute  a 
desired  future  location  for  the  vehicle  (point  B in  Figure  19)  relative  to  the  vehicle’s 
current  location  (point  A)  and  to  the  tangent  point  (point  T).  This  can  be  done  as  fol- 
lows. From  Figure  19: 


s = a tanA0 


A0  = tan~^  — 


(43) 


a 


Combining  Equations  (42)  and  (43): 


A0  = tan  '(r^tan^)) 


(44) 


Equation  (44)  gives  the  direction  A0  of  the  desired  point  B relative  to  the  direction  of 
the  tangent  point  7.  The  control  approach  is  then  to  continuously  measure  0 and  servo 
the  heading  vector  in  the  direction  A0.  This  approach  has  the  advantage  of  observing  a 
visible  point  (the  tangent  point)  and  servoing  toward  a near-by  point. 


To  determine  the  rate  at  which  to  servo  the  heading  vector,  we  derive  the 


following.  From  Equation  (44): 


Taking  the  derivative  with  respect  to  time: 

1 d(A(^\  V 


or 


4(A0) s cos^A0  ^ 

dt  he  cos^0 


(45) 
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Here  ^ is  the  optical  flow  of  the  tangent  point  in  the  0 direction.  The  value  0 can  be 
directly  measured  in  the  camera,  while  the  value  A0  is  computed  from  Equation  (44). 
The  values  j and  h,  are  a.  priori  known  constants.  Thus  the  only  measurements  needed 

for  computing  are  visual,  i.e.,  they  can  be  extracted  from  the  image. 

A final  scheme  that  can  be  useful  for  road  following  is  depicted  in  Figure  21.  The 
goal  is  to  servo  the  heading  vector  h so  as  to  maintain  the  vehicle  at  a constant  dis- 
tance 5 from  the  edge  of  the  road.  The  method  here  is  to  consider  an  imaginary  line  on 
the  ground  which  is  perpendicular  to  the  heading  vector  (and  to  the  camera  y-axis)  and 
is  at  a constant  distance  d from  the  vehicle.  We  measure  the  angle  0 between  the  X- 
axis  and  the  vector  from  the  camera  to  the  point  of  intersection  of  the  imaginary  line 
and  the  road  edge  (point  I in  Figure  21).  The  distance  5 from  the  road  edge  to  the 
heading  vector,  measured  along  the  imaginary  line,  is 

d 

^ = — r- 
tan0 

If  the  heading  vector  is  servoed  so  as  to  maintain  the  angle  0 constant  then,  because  d 
is  a constant,  5 will  also  be  maintained  at  a constant  value. 

Figure  22  shows  how  this  imaginary  line  can  be  determined  from  a tilt  angle  5, 
measured  in  the  camera  yz -plane.  From  Figure  22: 


This  equation  shows  how  the  value  of  5,  which  can  be  measured  in  the  image,  defines 
the  distance  d of  the  imaginary  line. 

From  Figure  21,  we  see  that  the  angle  0 should  always  be  smaller  than  the  tangent 
angle  07  if  the  road  edge  is  on  the  left  of  the  vehicle,  and  greater  than  0^  for  a road 
edge  on  the  right.  Since  0^  is  continuously  changing  for  a non-circular  road,  we  may 
want  to  have  several  imaginary  lines  at  distances  di,--,dn,  and  store  several 
corresponding  values  0i,---,0,.  These  distances  correspond  to  several  tilt  angles 

This  last  control  approach  is  the  only  one  in  this  paper  that  requires  information 
about  the  road  at  points  other  than  the  tangent  point.  We  discuss  this  kind  of  approach 
to  indicate  that  the  approaches  based  on  the  tangent  point  can  be  combined  with  other 
approaches  in  developing  robust  control  algorithms. 

5.  CONCLUSION 

This  paper  has  presented  a new  approach  to  vision  and  control  for  road  following. 
The  approach  is  based  on  an  analysis  of  the  geometric  and  motion-related  relationships 
and  constraints  for  road  following.  This  analysis  builds  upon  the  visual  field  theory  pre- 
viously developed  by  the  authors.  From  this  initial  analysis,  we  derived  visual  cues  and 
control  approaches  for  road  following. 
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We  showed  that,  in  principle,  the  only  road  feature  necessary  for  following 
curved,  convex  roads  is  the  position  of  the  tangent  point  on  the  road  edge  and  perhaps 
its  optical  flow.  In  practice,  larger  portions  of  the  road  may  have  to  be  extracted  in 
order  to  reliably  find  the  tangent  point.  We  also  showed  that  fast,  simple  control 
approaches  are  possible  that  directly  use  measured  image  quantities. 

The  partial  control  schemes  presented  in  this  paper  have  not  been  implemented 
yet.  Current  and  future  work  will  be  directed  towards  implementing  control  algorithms 
that  use  the  approaches  suggested  in  this  paper  [12].  Issues  such  as  the  dynamics  of  the 
vehicle,  sensitivity,  stability,  robustness,  and  timing  delays  must  be  considered  when 
developing  control  algorithms  for  real  vehicles. 

Another  area  for  future  work  is  to  extend  the  ideas  in  this  paper  to  other  types  of 
roads.  Issues  that  will  need  to  be  addressed  include  concave  roads,  roads  with  two 
boundaries,  where  the  left  and  right  boundaries  are  alternatively  convex,  and  roads  that 
are  straight  (not  curved).  Of  course,  to  follow  real-world  roads,  the  ideas  in  this  paper 
will  have  to  be  integrated  with  systems  that  find  road  edges  in  real  images  of  highways 
and  city  and  dirt  roads. 
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Figure  1:  Road  Following,  (a)  3-D  (b)  Top  View 
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Figure  2:  Coordinate  System  Fixed  to  Camera 
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Figure  5:  Overall  Description  of  Syste 


Figure  6:  Top  View 
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Figure  7:  Image  Obtained  from  Camera  Mounted 
Vehicle 
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Figure  9:  Section  of  a Zero  Flow  Cylinder 
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Filiure  11:  Seclion  of  Zero  Flow  Cylinder 


Figure  12:  Section  of  Zero  Flow  Cylinder  as  a Function 
of  Time 
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Figure  13:  Circular  Edge:  Top  View 
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Figure  15:  Four  Wheeled  Vehicle  with  Front  Wheel 
Steering 
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Figure  16:  Turning  Rate  Analysis 
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Figure  17:  Road  Following: 

(a)  Increasing  Radius  of  Curvature 

(b)  Decreasing  Radius  of  Curvature 
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Figure  18:  Change  in  Curvature:  Analysis 
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Figure  19:  Road  Following  - Analysis 
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Figure  20:  Road  Following  - Analysis 
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Figure  21:  Road  Following  - Analysis 
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Figure  22:  Road  Following  - Analysis 
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